Video colorization based on optical flow and edge-oriented color propagation

We propose a novel video colorization method based on sparse optical flow and edge-oriented color propagation. Colorization is a process of adding color to monochrome images or videos. In our video colorization method, it is assumed that key frames are appropriately selected out of a grayscale video stream and are properly colorized in advance. Once key frames are colorized, our method colorizes all the remaining grayscale frames automatically. It is also possible to colorize key frames semi-automatically by our method. For colorizing a grayscale frame between a pair of colorized key frames, sparse optical flow is computed first. The optical flow consists of reliable motion vectors around strong feature points. The colors around feature points of key frames are then copied to the grayscale frame according to the estimated motion vectors. Those colors are then propagated to the rest of the grayscale frame. Colors are blended appropriately during the propagation process. A pair of accuracy and priority measures is introduced to control how the color propagation proceeds. To successfully propagate colors, it is important not to wrongly spread colors across edges. For this purpose, a set of neighboring pixels is adaptively selected not to include edge-like areas and thus not to spread colors across edges. To evaluate effectiveness of our method, image colorization and video colorization were performed. Experimental results show that our method can colorize images and videos better than previous methods when there are edges. We also show that the proposed method enables us to easily modify colors in colored video streams.